Mark the hardcoded hybrid executors as deprecate and multi-exec as stable#46944
Conversation
Multiple Executor Configuration (aka hybrid executors) has been released for over half a year and can be marked stable. This gives users an option to migrate to from the old hardcoded hybrid executors (e.g CeleryKubernetesExecutor)
These executors will be removed in Airflow 3.0. Mark them as deprecated so users are aware and have time to migrate. - A deprecation warning will be printed when the executors are loaded - Documentation has been updated to note that they are deprecated - A news fragment added as well.
o-nikolas
requested review from
XD-DENG,
ashb,
hussein-awala,
kaxil,
pierrejeambrun and
potiuk
as code owners
boring-cyborg
bot
added
area:Executors-core
|
CC: @jedcunningham @jscheffl @amoghrajesh PR to mark the static executors as deprecated in Airflow core and mark multi-exec as stable. Re: https://lists.apache.org/thread/qmdf5dx71lzcvf4fd7qyc9gq0z0j2jqr |
Co-authored-by: Jed Cunningham <66968678+jedcunningham@users.noreply.github.com>
Co-authored-by: Jed Cunningham <66968678+jedcunningham@users.noreply.github.com>
|
Getting the tests to ignore the deprecation warning at the right time has proven very annoying/tricky. The last stubborn one is related to one instances of the warning not being thrown, but it seems that some underlying failure is actually happening, but I'm not sure why it's happening on this branch that should be otherwise stable. And I also cannot produce it locally. From the test failure logs: |
We already have coverage for this behaviour with the other executor
We do need to actually deprecate them in the providers code don't we? |
newsfragments/46944.significant.rst
Outdated
| @@ -0,0 +1,3 @@ | |||
| Statically-coded executors are now deprecated | |||
There was a problem hiding this comment.
We need here also announcement that hybrid executor feature is now stable.
There was a problem hiding this comment.
Good call. I've updated the newsfragment to make that more clear!
That one is less of a AF3 time crunch as it can be done at any time, and I actually also think it's fine to allow the provider to improve and support those executors into the near future. Since many people will continue to use them on AF2 for a long time to come. But yes, eventually they will need to be deprecated there as well as the user base shrinks. |
|
FYI I assume the failed CI run goes away on re-base... or as it is on v2-10-test you might need to cherry-pick #46980 Note I just realize that the PR target is v2-10-stable, I think this should rather be directed to v2-10-test! |
o-nikolas
deleted the
onikolas/v2-10-stable/deprecate-static-hybrid-execs
branch
Yupp, we should cherry-pick your change into v2-10-test
Also, yup, I accidentally targeted the wrong branch 🤦 I just chatted with @jedcunningham about it today, he's going to handle the mismatch during the next release. |
|
@jscheffl I added the backport-to-v2-10-test label to your PR to have it cherry picked back |
7 participants
In preparation of deprecating support in Airflow core for the old hardcoded hybrid executors in V3, mark them as deprecated and emit warnings when they are loaded. Also mark multi exec as stable to give users somewhere to migrate towards.
To broadcast the message of their deprecation this PR:
- Adds a deprecation warning when these old hybrid executors are loaded
- Documentation has been updated to note that they are deprecated and provide steps for migration.
- A news fragment was added as well.
The warning message:
Rendered docs:
^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named
{pr_number}.significant.rstor{issue_number}.significant.rst, in newsfragments.